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March  15, 1990 

The  Honorable  Sam  Nunn 
Chairman,  Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Les  Aspin 
Chairman,  Committee  on  Armed  Services 
House  of  Representatives 

The  Composite  Health  Care  System  (chcs)  is  a  state-of-the-art,  inte¬ 
grated  medical  information  system  the  Department  of  Defense  is  consid¬ 
ering  for  its  767  medical  treatment  facilities  worldwide.  \As  required  by 
the  National  Defense  Authorization  Act  for  Fiscal  Yeai>  1988  and  1989, 
we  have  been  monitoring  Defense’s  operational  test  and  evaluation 
(ot&e)  of  this  system.  Defense  plans  to  use  ot&e  results  and  related  cost/ 
benefit  analyses  to  decide  whether  chcs  should  be  procured  and  how  it 
should  be  deployed.  That  decision  is  scheduled  for  October  1990. 

The  Congress  established  specific  chcs  operational  testing  and  reporting 
requirements  in  the  act.  It  requires  that  Defense  conduct  ot&e  at  no 
fewer  than  six  sites  and  report  the  results  to  the  Senate  and  House 
Armed  Services  Committees.  According  to  a  Defense  directive,  ot&e’s 
purpose  is  to  ensure  that  only  operationally  effective  and  suitable  sys¬ 
tems  are  deployed.  Such  tests  are  needed  to  reduce  acquisition  risks  and 
ensure  that  systems  meet  technical  and  operational  requirements. 


Our  objectives  were  to  (1)  determine  whether  Defense  will  be  able  to  test 
and  evaluate  chcs  adequately  before  its  planned  procurement/deploy¬ 
ment  decision,  (2)  identify  Defense’s  latest  cost  and  funding  estimates, 
and  (3)  evaluate  the  reasonableness  of  Defense’s  projected  benefits.  We 
visited  Fort  Knox,  which  serves  as  Defense’s  test  site  for  new  software 
development,  and  6  of  1 1  OT&E  sites;  met  with  Defense  and  contractor 
representatives;  and  reviewed  government  and  contractor  documents. 

We  conducted  our  evaluation  from  July  1989  to  March  1990,  in  accord¬ 


ance  with  generally  accepted  government  auditing  standards  ^Appendix 
I  details  our  objectives,  scope,  and  methodology.  O,  ,  . ,  3 


This  status  report  discusses  Defense’s  progress  in  the  ot&e  phase  of  this 
system’s  development.  Our  final  report,  as  required  by  legislation,  will 
be  issued  30  days  after  the  Armed  Services  Committees  receive 
Defense's  report  on  the  results  of  ot&e. 
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Results  in  Brief 


Defense  faces  a  difficult  task  in  completing  an  adequate  ot&e  before  its 
planned  October  1990  procurement/deployment  decision.  As  of  Febru¬ 
ary  1990,  Fort  Knox  was  the  only  chcs  site  where  a  fully  integrated  sys¬ 
tem  was  operational.  Defense’s  current  plans  are  to  deploy,  test,  and 
evaluate  fully  the  chcs  system  at  six  additional  ot&e  sites  before  making 
its  decision.  These  sites  are  to  become  fully  operational  between  April 
and  June  1990.  This  will  permit  4  to  6  months  of  operation  and  evalua¬ 
tion  before  October  1990  instead  of  the  8  months  Defense  had  said,  in 
February  1989,  were  needed.  Defense  officials  are  confident  this  will 
provide  sufficient  time  to  test  the  system  and  determine  if  it  will  meet 
technical  and  operational  requirements.  We  believe  Defense’s  current 
test  schedule  is  extremely  tight  and  leaves  little  room  for  slippage. 
Defense  will  need  to  monitor  the  ot&e  closely  to  ensure  that  adequate 
information  is  obtained  and  that  its  test  and  evaluation  is  not  driven  by 
Defense’s  desire  to  complete  ot&e  by  October  1990. 

According  to  Defense’s  latest  estimate,  made  in  October  1989,  life-cycle 
costs  to  deploy  CHCS  to  its  767  medical  facilities  will  be  about  $1.6  bil¬ 
lion,  or  about  $600  million  more  than  the  $1.1  billion  congressional  ceil¬ 
ing.  Most  of  this  increase,  about  $436  million,  represents  a  Defense 
decision  to  extend  the  life  of  the  system  by  6  years.  Defense  plans,  in  its 
fiscal  year  1991  budget,  to  request  that  the  Congress  raise  the  ceiling  to 
$1.6  billion  to  cover  all  chcs  costs. 

The  chcs  program  office  estimates  that,  based  on  current  congressio¬ 
nally-approved  funding  of  $740  million,  Defense  will  be  able  to  deploy 
the  system  to  only  about  one-half  of  its  767  hospitals  and  clinics.  How¬ 
ever,  even  these  estimates  on  deployment  may  be  optimistic  because  the 
Defense  Inspector  General  reported,  in  1989,  that  about  $27  million,  that 
were  approved  for  Army  and  Navy  use  in  deploying  chcs.  have  been 
reprogrammed  for  other  purposes.  We  have  not  analyzed  the  appropri¬ 
ateness  of  this  reprogramming. 

Defense’s  projected  dollar  benefits  for  chcs  total  more  than  $2  billion. 
The  benefits  are  based  on  deployment  to  all  767  medical  facilities. 
Defense’s  CHCS-benefit  study  showed  that  about  96  percent  of  the  bene¬ 
fits  expected  from  chcs  are  expected  to  occur  in  the  champus  program. 
These  projections  assume  that  chcs  will  improve  the  availability  and 
timeliness  of  patient  information,  reduce  unnecessary  repeat  visits,  and 
eliminate  duplicate  tests.  According  to  Defense,  this  will  allow  physi¬ 
cians  and  nurses  more  time  to  treat  additional  patients.  Thus,  some 
patients  who  are  now  referred  to  civilian  medical  facilities  under 
champus  would,  instead,  be  treated  at  a  military  facility. 
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We  have  concerns  as  to  whether  Defense  will  be  able  to  realize  the  pro¬ 
jected  champus  benefits.  While  chcs  may  allow  facilities  to  treat  more 
patients,  current  champus  regulations  allow  beneficiaries  to  get  outpa¬ 
tient  care  from  civilian  hospitals  and  physicians  without  Defense’s 
approval  and  regardless  of  whether  the  military  facility  has  excess 
capacity.  Additionally,  the  benefit  study  did  not  consider  restrictions  on 
the  number  of  patients  specialists  may  treat  during  a  given  period)  For 
example,  the  Naval  Medical  Command  limits  obstetricians  to  20  deliv¬ 
eries  a  month.  In  the  area  served  by  a  certain  Naval  hospital,  this 
change  alone  increased  the  average  monthly  CHAMPUS-paid  obstetrics 
cases  from  47  to  347. 

Defense  agrees  that  estimating  chcs  benefits  is  difficult  and  is  in  the 
process  of  refining  the  cost/benefit  analysis  that  will  be  submitted  to  the 
Congress  at  the  conclusion  of  ot&e. 


Background 


/ 


chcs  is  a  state-of-the-art,  integrated,  medical  information  system 
Defense  is  testing  for  implementation  at  its  medical  treatment  facilities. 
On  the  leading  edge  of  technology  and  beyond  the  capabilities  of  sys¬ 
tems  commercially  available,  chcs  is  designed  to  improve  the  timeliness, 
availability,  and  quality  of  patient-care  data.  It  will  replace  manual  and 
automated  information  systems  now  supporting  Defense  medical  treat¬ 
ment  facilities.  At  individual  hospitals,  it  will  integrate  the  functional 
work  centers  of  inpatient  and  outpatient  care  facilities,  patient  adminis¬ 
tration,  patient  appointment  and  scheduling,  nursing,  laboratory,  phar¬ 
macy,  radiology,  and  clinical  dietetics,  chcs  is  intended  to  provide 
physicians  with  immediate  access  to  patient  medical  records. 

Congress  established  specific  chcs  operational  testing  and  reporting 
requirements  in  the  fiscal  year  1988-1989  National  Defense  Authoriza¬ 
tion  Act.  The  act  requires  that  Defense 

conduct  ot&e  at  no  fewer  than  six  sites,  and 

submit  a  report  to  the  Senate  and  House  Armed  Services  Committees 
that  evaluates  ot&e  results,  analyzes  chcs  costs  and  benefits,  and  con¬ 
tains  a  deployment  plan  based  on  the  cost/benefit  analysis. 

Office  of  Management  and  Budget  Circular  A- 109  and  Defense  guidance 
require  that  ot&e  be  conducted  in  an  operationally-realistic  environ¬ 
ment.  ot&e  should  determine  whether  a  system  (1)  will  meet  and  satisfy 
operational  requirements  and  mission  needs,  and  (2)  can  be  deployed  on 
schedule.  The  Office  of  Health  Systems  Evaluation,  within  the  Office  of 
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the  Secretary  of  Defense,  is  conducting  the  ot&e  for  chcs.  Details  on  ot&e 
in  major  acquisitions  are  discussed  in  appendix  II. 

In  early  1988,  after  evaluating  system  demonstrations  and  proposals 
from  competing  contractors,  Defense  awarded  Science  Applications 
International  Corporation  a  contract  to  develop,  test,  deploy,  and  sup¬ 
port  chcs.  Fort  Knox  served  as  the  test  site  at  which  the  contractor 
demonstrated  its  ability  to  implement  chcs  and  continues  to  operate  as  a 
test  site  for  new  software.  As  part  of  the  contract,  Defense  first  planned 
to  deploy  and  test  the  system  at  nine  additional  sites — seven  in  the  con¬ 
tinental  United  States,  one  in  Europe,  and  one  in  the  Pacific. 

Because  of  our  concern  that  the  nine  test  sites  did  not  fairly  represent 
Defense  medical  facilities,  which  range  in  size  from  small  clinics  to  hos¬ 
pitals  with  over  500  beds,  Defense  expanded  the  number  of  ot&e  sites  in 
February  1989. 1  Two  large  (Walter  Reed  and  Bethesda)  and  two  small 
(Carswell  and  Shaw)  hospitals  were  added.  At  that  time,  Defense 
expected  to  complete  or&E  at  Fort  Knox  and  the  13  sites  by  August  1990. 

In  February  1990,  as  a  result  of  schedule  slippage  and  in  response  to 
concerns  we  raised  during  this  review,  Defense  postponed  its  procure¬ 
ment/deployment  decision  until  October  1990.  In  addition,  because  of 
physical  limitations  in  the  Bethesda  computer  facility  and  budgetary 
constraints  relating  to  Carswell,  Defense  dropped  these  two  sites  from 
ot&e  testing.  The  12  currently  active  test  sites  are  listed  in  appendix  III. 
Further,  Defense  decided  to  focus  its  test  and  evaluation  efforts  at  seven 
sites — Fort  Knox  and  six  additional  ot&e  sites:  Charleston,  Eglin,  Jack¬ 
sonville,  Nuernberg,  Sheppard,  and  Tripler. 

During  the  early  stages  of  ot&e,  Defense  has  used  its  Fort  Knox  site  to 
develop  and  test  chcs  software  and  formulate  an  approach  to  determine 
whether  anticipated  benefits  will  be  realized.  Beginning  in  April  1990, 
Defense  plans  to  begin  deployment  and  operation  of  chcs  at  the  six  other 
ot&e  sites.  Additionally,  Defense  will  be  testing  at  Shaw,  primarily,  to 
determine  the  cost-effectiveness  of  deploying  at  small  facilities. 


1  Medical  ADP  Systems:  Composite  Health  Care  System  Operational  Tests  Extended  (GAO/ 
IMTEC-WMO,  Apr.  10, 1955)  -  -  - 
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Defense  Faces  a 
Difficult  Task  in 
Completing  an 
Adequate  OT&E  by 
October  1990 


Table  1:  CHCS  Test  Site  Implementation 
Schedule  as  of  February  1990 


Originally  scheduled  for  completion  in  September  1989,  ot&e  is  now 
scheduled  to  be  finished  in  October  1990.  In  February  1989,  when 
Defense  first  extended  ot&e,  system  implementation  and  user  training 
were  expected  to  be  completed  at  13  test  sites  by  December  1989.  That 
schedule  would  have  allowed  about  8  months  to  complete  the  ot&e — 3 
months  for  system  stabilization  and  5  months  to  complete  testing,  evalu¬ 
ation,  and  reporting.  Defense  officials  maintained  that  this  time  was 
needed  to  reduce  the  government’s  risk,  as  the  additional  time  would 
allow  more  time  to  test  the  system  and  allow  for  routine  use  at  the  13 
sites  before  a  deployment  decision.  They  also  maintained  that  gathering 
data  from  more  experienced  users  would  improve  their  ability  to  demon¬ 
strate  system  benefits. 

As  shown  in  table  1,  Defense  has  implemented  the  integrated  system  at 
Fort  Knox.  Defense  plans  to  have  the  integrated  system  implemented 
and  to  test  and  evaluate  it  for  4  to  6  months  at  six  additional  sites  by 
October  1990. 


Test  site 

Hospital 

beds 

Clinical 

visits 

Scheduled 
implementation 
of  integrated 
system 

Fort  Knox  [Kentucky]3 

195 

621,423 

May  1989 

Charleston  [South  Carolina] 

184 

359,576 

Apr  1990 

Eglin  [Florida] 

145 

446,431 

Apr  1990 

Jacksonville  [Florida] 

178 

570,706 

Apr  1990 

Nuernberg  [West  Germany] 

142 

519,309 

May  1990 

Sheppard  [Texas] 

135 

247,285 

May  1990 

Tripler  [Hawaii] 

479 

1,573,369 

Jun  1990 

Shaw  [South  Carolina] 

40 

174,006 

Sep  1990 

Eisenhower  [Georgia] 

384 

650,034 

Dec  1990 

Keesler  [Mississippi] 

295 

442,368 

Dec  1990 

Walter  Reed  [Washington,  D.C.] 

886 

1,222,767 

Feb  1991 

LeJeune  [North  Carolina] 

170 

364,264 

Feb  1991 

aTest  site  for  development  and  testing  of  software.  Implementation  is  the  actual  date 


As  currently  planned,  the  integrated  system  will  not  be  fully  imple¬ 
mented  at  the  largest  facility,  Walter  Reed,  and  will  be  implemented  for 
1  month,  prior  to  the  completion  of  ot&e,  at  the  smallest  facility — Shaw. 
As  stated  earlier,  Defense  added  these  test  sites  to  make  its  ot&e  more 
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Some  CHCS  Software 
Will  Not  Be  Tested 


representative  of  its  medical  facilities.  Testing  at  larger  military  hospi¬ 
tals,  where  the  greatest  difficulty  is  expected,  would  provide  an  oppor¬ 
tunity  to  (1)  fully  stress  the  system,  (2)  validate  the  assumption  that 
those  facilities  will  derive  the  greatest  benefits  from  the  system,  and  (3) 
measure  Defense’s  and  the  contractor’s  ability  to  implement  the  system 
at  large  facilities.  A  full  operational  test  at  the  small  site  (Shaw)  is  not 
planned;  instead,  Defense  will  test  chcs  at  Shaw,  primarily,  to  determine 
the  cost-effectiveness  of  deploying  chcs  to  small  facilities.  The  results  of 
this  test  is  important  because  these  facilities  represent  about  two-thirds 
of  the  hospitals  in  the  military  health  care  system. 

Defense’s  current  approach  to  testing  a  large  hospital  includes  (1)  a 
fully  operational  test  at  Tripler  to  stress  system  software  and  hardware, 
and  (2)  testing  the  contractor’s  ability  to  perform  site  preparation  and 
initiate  operational  testing  at  Walter  Reed.  Tripler  ranks  among  the  10 
largest  military  hospitals  in  terms  of  hospital  beds  and  clinical  visits. 
ot&e  at  this  site  will  include  supporting  personnel  from  all  three  military 
services  and  nine  clinical  sites  in  Hawaii.  Because  of  the  additional 
requirements  imposed  by  the  nine  clinics,  the  Tripler  test  should  be  rep¬ 
resentative  of  the  very  largest  medical  facilities.  Defense  officials 
believe  that  these  requirements  will  sufficiently  stress  the  system  and 
reduce  the  risk  of  deployment  to  the  remaining  large  hospitals. 

Defense’s  current  schedule  has  reduced  the  time  to  perform  and  evalu¬ 
ate  the  operational  tests  from  8  months  to  4-6  months.  Compressing  the 
time  to  test  and  evaluate  chcs  system  performance  adds  risk  to 
Defense’s  ability  to  make  a  sound  deployment  decision  and  will  require 
close  Defense  monitoring. 


Defense’s  current  plans  show  that  about  38  percent  of  system  software 
will  not  be  developed  and  deployed  to  the  test  sites  by  October  1990. 
This  untested  software  will  include  some  capabilities  currently  desig¬ 
nated  as  high  priority  by  the  Surgeons  General  of  the  three  services.  For 
example,  the  human  resource  management  component,  which  provides 
staffing,  scheduling,  teaching,  and  orientation  for  nurses  will  be 
untested,  as  will  the  transfusion-services-management  component, 
which  automates  blood-test  ordering,  results  entry,  documentation,  and 
inventory.  Defense,  as  of  February  1990,  did  not  have  specific  dates  for 
testing  this  software. 

However,  the  program  office  estimates  that  the  system  software,  which 
will  have  been  tested  by  October  1990  will  provide  about  87  percent  of 
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the  projected  dollar- valued  benefits  and  about  79  percent  of  the  system 
capabilities  that  the  Surgeons  General  identified  as  high  priority. 
Although  we  have  not  evaluated  the  accuracy  of  these  estimates,  we 
found  that  the  system  in  operation  at  Fort  Knox  and  scheduled  for 
deployment  to  the  cjt&e  test  sites  fully  integrates  such  activities  as 
patient  appointment  and  scheduling,  physician  order  entry,  laboratory, 
radiology,  and  nursing.  These  activities  cover  a  substantial  portion  of  a 
hospital’s  activities. 

Defense  has  coordinated  software  development  and  deployment  with 
the  Surgeons  General,  and  the  deployment  of  the  remaining  high-prior¬ 
ity  software  is  being  discussed.  Because  of  changes  in  the  composition  of 
the  Surgeons  General,  there  has  been  some  shift  in  the  order  this 
remaining  software  should  be  developed. 


Estimated  Costs 
Exceed  Congressional 
Ceiling 


Defense  estimates  chcs  life-cycle  costs  for  full  deployment  to  767  medi¬ 
cal  facilities  at  $1.6  billion,  or  $500  million  more  than  the  $1. 1-billion 
congressional  ceiling.  The  congressional  ceiling  for  cues  was  established 
in  the  Department  of  Defense  Appropriations  Act  for  fiscal  year  1987 
and  has  remained  in  effect  in  succeeding  appropriations  acts. 


Defense  guidance  defines  life-cycle  costs  as  contract  and  in-house 
costs — development,  procurement,  operation,  support,  and  where  appli¬ 
cable,  disposal.  Table  2  shows  Defense’s  estimate  of  total  life-cycle  costs 
through  fiscal  year  2002 — the  expected  life  of  the  system. 


Table  2:  Defense’s  Estimate  of  Total 
CHCS  Life-Cycle  Costs  in  Fiscal  Year 
1986  Dollars 


(Dollars  in  millions) 

Prime 

Contract 

costs 

Other 

Government 

costs* 

Total  cost 

Demonstration  projects 

$14.3 

$21.3 

$35.0 

Acquisition/contractor  competition 

75.3 

21.0 

963 

Testing/deployment  operations 

858.1 

139  3 

997  4 

Subtotal 

$947.7 

$181.6 

$1,129.3 

Extend  life  cycle6 

381.7 

52.9 

434  6 

Total 

$1,329.4 

$128.0 

$1,563.9 

includes  such  costs  as  government  personnel,  increased  power  and  air  conditioning,  and  system 
upgrades. 

bln  a  March  1988  project  review,  Defense  decided  to  extend  the  life  cycle  for  CHCS  from  10  to  15  years 
and  provide  for  mid-life,  technology  enhancements. 
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chcs  life-cycle  costs  for  full  deployment  are  expected  to  be  $500  million 
more  than  the  $1.1 -billion  congressional  ceiling.  While  we  have  not  eval¬ 
uated  the  $500-million  increase,  Defense  states  that  it  is  primarily  the 
result  of  a  decision  to  extend  the  project’s  life  cycle  by  5  years.  As  part 
of  its  fiscal  year  1991  budget  request  to  the  Congress,  the  program 
office  is  requesting  that  the  ceiling  be  raised  to  $  1 .6  billion  to  cover  all 
costs. 


Projected  Funding 
Inadequate  for 
Worldwide 
Deployment 


As  of  fiscal  year  1990,  the  Congress  has  approved  $740  million  for  the 
deployment  of  chcs  through  fiscal  year  1997.  The  $740  million  is  based 
on  approved-service  resources  included  in  the  Fiscal  Year  1990-1994 
Program  Objective  Memorandum  and  projected  funding  through  fiscal 
year  1997.  At  that  funding  level,  Defense  now  estimates  that  it  will  be 
able  to  deploy  chcs  to  only  about  one-half  of  its  facilities.  These  facilities 
support  about  57  percent  of  the  services’  inpatient  care  and  39  percent 
of  outpatient  care.  Defense  believes  that  the  current  funding  level  of 
$740  million  would  reduce  benefits  by  about  $500  million. 

However,  it  is  questionable  whether  Defense  has  $740  million  available. 
The  Army  and  Navy  have  reprogrammed  about  $27  million  in  chcs 
funding  to  other  areas.  As  of  February  1990,  for  fiscal  years  1992 
through  1994,  the  Army  reprogrammed  $12  million  in  chcs  procurement 
funding.  Also,  the  Navy  reprogrammed  $15  million  it  had  planned  to 
spend  for  chcs  procurement  in  fiscal  years  1993  and  1994.  Approved 
Navy  procurement  funding  for  these  years  was  intended  to  deploy  chcs 
to  27  hospitals  and  clinics.  We  have  not,  however,  analyzed  the  appro¬ 
priateness  of  these  reprogrammings. 


Expected  B6nefitS  Arc  Defense’s  projected  dollar  benefits  for  chcs  through  fiscal  year  1997 
TYf-f*  1+  4-  I?  +  total  more  than  $2  billion.  The  benefits  assume  full  deployment  to  all 

DlttlCUlt  tO  Estimate  767  facilities  and  largely  depend  on  reducing  the  costs  of  the  Civilian 

Health  and  Medical  Program  of  the  Uniformed  Services  (champus). 
champus  pays  for  health  care  costs  for  families  of  uniformed  services 
members  who  are  unable  to  get  care  through  a  military  hospital  or  clinic 
and  receive  medical  services  from  private  providers. 

As  shown  in  table  3,  about  95  percent  of  the  projected  chcs  benefits  are 
expected  to  occur  in  champus.  These  projections  assume  that  chcs  will 
improve  the  availability  and  timeliness  of  patient  information,  reduce 
unnecessary  repeat  visits,  and  eliminate  duplicate  tests,  thereby, 
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increasing  hospital  capacity.  According  to  Defense,  this  will  allow  mili¬ 
tary  physicians  and  nurses  more  time  to  treat  additional  patients.  Thus, 
patients  who  now  receive  care  from  civilian  medical  facilities  under 
champus  could,  instead,  be  treated  at  a  military  treatment  facility. 


Table  3:  CHCS  Life-Cycle  Benefits  in 
Fiscal  Year  1986  Dollars 


Dollars  in  millions 

Category 

Methodology 

Savings 

Percent 

of 

savings 

Inpatient  care 

CHAMPUS  conversion 

$1,216.8 

58.9 

Outpatient  care 

CHAMPUS  conversion 

742.2 

35.9 

Other  direct  cost  offsets 

Spoilage  reduction 

Malpractice  reduction 

Medical  claims  increase 

Eligibility  enrollment  expense 
reduction 

28.0 

1.4 

Active-duty  time  savings 

Decreased  waiting  time 

Decreased  prescription  reactions 

66.2 

3.2 

Non  active-duty  time 
savings 

Decreased  waiting  time 

Decreased  prescription  reactions 

13.2 

.6 

Total 

$2,066.4 

100.0 

Defense  developed  a  benefits  analysis  model  to  project  potential  savings 
that  would  accrue  if  some  patients  now  being  treated  at  private  facilities 
were,  instead,  treated  in  a  military  facility.  This  model  projects  that 
chcs  efficiencies  will  reduce  the  average  length  of  an  inpatient  stay  by 
about  5  percent,  thereby  allowing  the  hospital  to  treat  more  patients. 

We  identified  weaknesses  in  Defense’s  benefits  study  that  raise  ques¬ 
tions  regarding  the  ability  of  Defense  to  achieve  the  full  extent  of  the 
projected  champus  savings — even  if  deployed  to  all  767  medical 
facilities. 

First,  the  projected  benefits  assume  that  hospitals  and  physicians  would 
be  more  efficient  after  chcs  is  implemented,  and,  thereby,  have  more 
time  to  treat  patients  who  would  otherwise  be  referred  to  champus. 
However,  champus  regulations  allow  care  from  civilian  hospitals  and 
physicians  without  Defense’s  approval  and  regardless  of  whether  the 
military  facility  has  excess  capacity.  Defense’s  projected  benefits  do  not 
consider  that  patients  can  obtain  inpatient  care  under  champus  even 
though  a  military  facility  could  treat  them.  For  example,  obstetrics  and 
gynecology  patients  within  the  Jacksonville  Naval  Hospital  area  are  not 
required  to  get  their  care  at  this  hospital  if  they  live  more  than  45  min¬ 
utes  driving  time  from  the  facility.  This  exemption  was  prompted  by 
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Conclusion 


highway  and  bridge  traffic  congestion  frequently  encountered  in  the 
Jacksonville  area.  For  safety  and  medical  reasons,  patients  in  outlying 
geographical  areas  routinely  use  the  champus  program  and  receive  care 
in  private  hospitals  closer  to  their  homes. 

Additionally,  the  projected  benefits  do  not  consider  restrictions  on  the 
number  of  patients  specialists  may  treat  during  a  given  period.  For 
example,  the  Naval  Medical  Command  limits  obstetricians  to  20  deliv¬ 
eries  a  month.  In  our  July  1989  report  on  champus2  ,  we  noted  that,  pri¬ 
marily  because  of  that  limitation,  average  monthly  deliveries  for 
obstetricians  at  the  San  Diego  Naval  Hospital  decreased  from  37  in  fiscal 
year  1985  to  18  in  fiscal  year  1987.  Thus,  total  average  monthly  deliv¬ 
eries  at  the  San  Diego  Naval  Hospital  decreased  from  about  400  to  200, 
while  average  monthly  CHAMPUS-paid  inpatient  obstetrics  cases  for  the 
area  served  by  that  hospital  ballooned  from  47  to  347.  According  to  the 
chcs  program  office,  this  restriction  is  being  assessed  to  determine  how 
it  will  affect  benefit  projections. 

Defense  is  going  to  have  difficulty  in  accurately  estimating  chcs  benefits 
by  October  1990  for  some  of  the  reasons  discussed  above,  and  because  it 
will  likely  take  several  years  of  experience  to  determine  accurately 
chcs’s  effect  on  champus.  Defense  is  aware  of  the  difficulty  and  is  in  the 
process  of  further  refining  its  estimates.  Its  updated  cost/benefit  analy¬ 
sis  is  to  be  included  in  the  report  it  plans  to  provide  to  the  Senate  and 
House  Armed  Services  Committees  at  the  conclusion  of  its  ot&e. 


chcs  has  experienced  delays  in  its  development  and  planned  or&E  sched¬ 
ule.  We  recognize  that  delays  with  the  development  of  a  state-of-the-art 
system,  such  as  chcs,  are  not  uncommon.  During  the  remainder  of  the 
ot&e,  Defense  plans  to  continue  implementing  chcs  at  its  test-site  facili¬ 
ties.  The  progress  made  at  these  sites  in  demonstrating  the  system’s  abil¬ 
ity  to  generate  benefits  will  be  critical  in  providing  an  adequate  basis  on 
which  to  base  future  deployment  decisions.  The  worst  thing  that  could 
happen  would  be  to  deploy  prematurely  and,  then,  find  problems  that 
preclude  the  system’s  meeting  its  goals. 

Defense’s  current  ot&e  plans  focus  on  demonstrating  that  chcs  can  be 
deployed  to  and  become  fully  operational  at  six  sites.  The  current  sched¬ 
ule  calls  for  4  to  6  months  of  operation  and  testing  at  these  sites  before 


2Defense  Health  Care:  Workload  Reductions  at  Military  Hospitals  Have  Increased  CHAMPUS  Costs 
(GAO/HRt»-89-47,  July  10,  ld86). 
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the  end  of  or&E  in  October  1990.  Meeting  the  objectives  set  forth  in  this 
schedule  are  critical  to  Defense  being  able  to  make  an  informed  deploy¬ 
ment  decision  and  meet  the  legislative  requirement  that  or&E  be  con¬ 
ducted  at  six  sites. 

In  our  previous  report,  we  recommended,  and  Defense  agreed,  that  or&E 
should  include  a  full  range  of  operational  environments,  including  large 
and  small  military  hospitals.  Defense’s  current  approach  to  testing  chcs 
at  larger  hospitals  involves  (1)  stressing  system  hardware  and  software 
capabilities  at  Tripler  and  (2)  testing  contractor  deployment  capabilities 
at  Walter  Reed.  The  cost-effectiveness  of  deploying  chcs  to  smaller  hos¬ 
pitals  will  be  tested  at  Shaw  for  1  month.  While  we  are  concerned  about 
whether  1  month  is  adequate  for  Shaw,  this  approach  appears  to  be  rea¬ 
sonable  and,  if  successful,  should  reduce  the  risk  of  a  deployment  to 
Defense’s  larger  and  smaller  hospitals. 

We  recognize  that  accurately  estimating  system  benefits  is  not  easy. 
Estimating  CHAMPUS  benefits  is  a  problem,  since  many  factors  are  at 
work  that  may  preclude  Defense  from  getting  good  information  on 
whether  CHAMPUS  patients  will  return  in  sufficient  numbers  to  achieve 
the  projected  benefits.  Further,  congressionally-approved  funding 
levels,  facility  deployment  schedules,  and  the  decisions  regarding  soft¬ 
ware  not  included  in  the  ot&e  will  also  have  an  impact  on  the  accuracy 
of  the  benefit  estimates.  Defense  agrees  that  it  will  be  difficult,  and  it  is 
in  the  process  of  further  refining  its  estimates.  The  cost/benefit  analysis 
required  to  be  included  in  the  ot&e  evaluation  report  should,  if  properly 
done,  provide  the  information  needed  to  make  an  informed  procure¬ 
ment/deployment  decision. 


We  discussed  the  contents  of  this  report  with  senior  Defense  officials, 
who  generally  agreed  with  our  findings.  We  have  incorporated  their 
comments  as  appropriate. 

We  are  sending  copies  of  this  report  to  the  Chairmen  of  the  House  and 
Senate  Committees  on  Appropriations;  the  Director,  Office  of  Manage¬ 
ment  and  Budget;  and  the  Secretary  of  Defense.  Copies  also  will  be  made 
available  to  other  interested  parties  upon  request. 


Page  11 


G A0  TMTEC-90-4Z  Defense's  Difficult  Tart 


B-220732 


This  work  was  performed  under  the  direction  of  Daniel  C.  White,  Special 
Assistant  to  the  Assistant  Comptroller  General,  who  can  be  reached  at 
(202)  275-4659.  Other  major  contributors  are  listed  in  appendix  IV. 


r Ralph  V.  Carlone 

Assistant  Comptroller  General 
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Objectives,  Scope,  and  Methodology 


The  National  Defense  Authorization  Act  for  Fiscal  Years  1988  and  1989 
requires  that  we  (1)  monitor  the  or&E  phase  and  related  chcs  acquisition 
activities  and  (2)  submit  a  report  to  the  Senate  and  House  Armed  Ser¬ 
vices  Committees  evaluating  crr&E  results  and  Defense’s  contract  award 
process  for  the  chcs  procurement/deployment.  This  status  report  is  one 
of  a  series  of  reports  that  we  have  issued  dealing  with  Defense’s  acquisi¬ 
tion  and  testing  of  this  system.1  As  required  by  the  act,  our  final  report 
on  chcs  or&E,  as  required  by  legislation,  will  be  issued  30  days  after  the 
Armed  Services  Committees  receive  Defense’s  report  on  the  results  of 
or&E. 


For  this  report,  our  objectives  were  to  (1)  determine  whether  Defense 
will  be  able  to  test  and  evaluate  chcs  adequately  before  its  planned  pro¬ 
curement/deployment  decision,  (2)  determine  Defense’s  latest  cost  and 
funding  estimates,  and  (3)  evaluate  the  reasonableness  of  Defense’s  pro¬ 
jected  benefits. 

We  visited  6  of  the  11  or&E  sites:  Eglin,  Eisenhower,  Jacksonville, 
Keesler,  Sheppard,  and  Walter  Reed.  We  also  visited  Science  Applica¬ 
tions  International  Corporation,  in  Falls  Church,  Virginia,  and  the 
Defense  Medical  Systems  Support  Center  (the  program  office  managing 
the  chcs  acquisition).  We  also  visited  the  Army  Medical  Center  at  Fort 
Knox,  Kentucky,  where  Science  Applications  International  Corporation 
is  conducting  the  initial  testing  for  new  chcs  software. 

We  worked  closely  with  senior  program  management  officials  to  (1)  dis¬ 
cuss  our  concerns  as  they  arose,  (2)  confirm  our  understanding  of  poten¬ 
tial  problems  and  their  implications  for  achievement  of  test  objectives, 
and  (3)  permit  them  to  respond  to  our  observations.  We  briefed  senior 


'In  previous  reports,  we  examined  issues  pertaining  to  Defense’s  acquisition  process.  See 

ADP  Systems:  Concerns  About  the  Acquisition  Plan  for  DOD’s  Composite  Health  Care  System  (GAO/ 

IMTEC-86-T2,  Mar.  31,  l£$6). 

ADP  Systems:  Concerns  About  DOD’s  Composite  Health  Care  System  Development  Contracts  (GAO/ 
EMTEC-STSS,  June  8, 1087). -  - - - 

Medical  ADP  Systems:  Composite  Health  Care  System  Operational  Test  and  Evaluation  Costs  (GAO/ 
IMTCC-8M8BR,  Jan.  2S,  m&i -  - 

Medical  ADP  Systems:  Composite  Health  Care  System  Acquisition-Fair,  Reasonable,  Supported 
(UAO/1MTECS8-126,  Mar.  A198S). 

Medical  ADP  Systems:  Analysis  of  Technical  Aspects  of  DOD's  Composite  Health  Care  System  (GAO/ 
IMTEC-88-U7,  July  11, 1985).  - - 

Medical  ADP  Systems:  Composite  Health  Care  System  Operational  Tests  Extended  (GAO/ 
IMTEC-8030 ,ApF.  10,  - 
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program  management  officials  during  our  review  and  incorporated  their 
views  where  appropriate. 
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Operational  Test  and  Evaluation  in  Major 
Defense  Acquisitions 


In  response  to  growing  congressional  concerns  about  the  risks  associated 
with  acquiring  complex  and  costly  medical  ADP  systems,  the  Assistant 
Secretary  of  Defense  (Comptroller)  in  1979  directed  that  the  chcs  acqui¬ 
sition  comply  with  Office  of  Management  and  Budget  Circular  A-109 
acquisition  guidelines.  These  guidelines  instruct  federal  agencies  on  how 
to  conduct  a  major  system  acquisition  and  minimize  risks  of  inadequate 
system  performance  and  excessive  costs.  The  circular  addresses  all 
aspects  of  the  acquisition  process.  Under  the  A-109  strategy,  a  full-scale 
test  of  a  system  is  conducted  to  determine  if  it  will  perform  effectively 
under  operational  conditions. 

The  director  of  ot&e  within  the  Office  of  the  Secretary  of  Defense  has 
the  responsibility  to  monitor  and  review  all  ot&e  within  Defense  (10 
U.S.C.  138).  Systems  to  be  tested  include  all  major  Defense  systems 
acquisitions  with  estimated  life-cycle  costs  greater  than  $1  billion  (10 
U.S.C.  2430).  chcs  is  a  major  system  acquisition  with  projected  life-cycle 
costs  of  more  than  $1  billion. 

ot&e  generally  seeks  to  determine  (1)  whether  a  system  will  satisfy  mis¬ 
sion  needs  and  is  suitable  for  use  by  typical  military  users  and  (2)  if 
Defense  and  the  contractor  developing  the  system  are  capable  of 
deploying  it  on  schedule.  Although  field  testing  of  weapons  systems  is 
the  primary  application  of  or&E,  Defense  believes  that  automated  infor¬ 
mation  systems,  such  as  chcs,  require  the  same  level  of  testing  as  do 
major  weapons  systems  to  determine  their  effectiveness  and  suitability 
in  the  environment  in  which  they  will  operate.  To  reduce  the  risk  of 
deploying  a  costly  medical  information  system  before  it  is  adequately 
developed  and  tested,  Defense  recognizes  that  chcs  must  be  tested  in  a 
realistic  operating  environment  before  the  system  can  be  deployed 
throughout  the  military  hospital  system. 
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CHCS  OT&E  Test  Site  Implementation  Schedule 
as  of  February  1990 


Test  site 

Hospital 

beds 

Scheduled 
implementation  of 
Site  prep  complete*  integrated  system 

Fort  Knox6 

195 

May  1989 

Charleston6 

184 

August  1988 

April  1990 

Eglinc 

145 

August  1988 

April  1990 

Jacksonville6 

178 

July  1988 

April  1990 

Nuernberg6 

142 

September  1988 

May  1990 

Sheppard6 

135 

July  1988 

May  1990 

Tripler6 

479 

October  1988 

June  1990 

Shaw 

40 

May  1990 

September  1990 

Eisenhower 

384 

October  1988 

December  1990 

Keesler 

295 

July  1988 

December  1990 

Walter  Reed 

886 

November  1989 

February  1991 

LeJeune 

170 

August  1988 

February  1991 

Bethesda6 

494 

Carswell6 

90 

“Includes  hardware  installation. 

bDefense  began  installing  and  testing  CHCS  in  1987  during  the  competition  stage  of  the  acquisition  It 
still  serves  as  Defense's  test  site  for  new  software  development. 

cln  February  1990,  Defense  decided  to  focus  its  OT&E  efforts  on  Fort  Knox  and  these  6  test  sites. 

a8ecause  of  physical  limitations  in  the  Bethesda  computer  facility  and  budgetary  constraints  relating  to 
Carswell,  Defense  dropped  these  two  sites  from  OT&E  testing. 
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